Strobe image composition method, apparatus, computer, and program product

ABSTRACT

There are disclosed a method, apparatus, computer, and program product for generating a strobe composite image from a plurality of frames of a moving image. A first frame is selected from the plurality of frames of the moving image. A plurality of second frames relating to the first frame are determined. Then, a strobe composite image is generated by superposing the plurality of second frames in accordance with a superposing manner.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Divisional of U.S. patent application Ser. No.10/664,878, filed Sep. 22, 2003, and is based upon and claims thebenefit of priority from the prior Japanese Patent Applications No.2002-288014, filed Sep. 30, 2002; and No. 2003-068968, filed Mar. 13,2003. The entire contents of these applications are incorporated hereinby reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method, apparatus, computer, andprogram product for generating a strobe composite image by superposingtime-serially obtained object images.

2. Description of the Related Art

An image processing technique for generating and displaying a compositeimage by extracting only an object portion from a moving image andsuperposing it on another image is effective to generate a strobecomposite image by superposing object images sensed at a plurality oftimes. In this case, if a user interface that allows the user tosuperpose a plurality of object images is not available, the user cannoteasily generate a strobe composite image, and must spend much time inoperations.

Conventionally, a technique for superposing a plurality of imagecomponents called layers each having a transparent portion has been usedto generate a strobe composite image by superposing a plurality ofobject images (for example, see reference “Adobe Premier 6.0 Manual,Chapter 7, pp. 281-294”). Initially, only object image regions areextracted from a moving image by an arbitrary method. In this process, achroma-key method that senses motion of an object in front of a uniformcolor background, and extracts object images by exploiting colorcomponent differences can be adopted. Then, the extracted object imageregions are copied to form a plurality of layers using a layer functionof image processing software, and a strobe composite image can beobtained by superposing these layers.

However, in order to create a strobe composite image by the conventionaltechnique, the user must manually create layers corresponding to frames,and must manually designate the superposing manner as a hierarchy uponcreating layers or must manually adjust the hierarchy of layers afterthe layers are created. Also, in the conventional technique, the usermust execute strobe composition by capturing an image which is toundergo strobe composition, and saving it in a file. For example, if itis impossible to photograph an object in front of a uniform backgroundlike in sport games, the chroma-key method cannot be used to extract anobject portion from an image. Therefore, a manual extraction process isrequired, thus taking much time until a strobe composite image iscreated.

It is often required to shorten the time period required from when animage is obtained until a strobe composite image is created, since thestrobe composite image is often used in comments of, e.g., the form of aplayer in a television live program. Such requirements cannot be metsince it takes much time to create a strobe composite image using theconventional technique. Therefore, a method of generating a desiredstrobe composite image within a short period of time is demanded.

BRIEF SUMMARY OF THE INVENTION

The present invention has as its object to provide an image compositionmethod, apparatus, computer, and program product, which can reduce theload on the user upon generating a strobe composite image, and canshorten the time required for an image generation process.

A method, apparatus, computer, and program product according to anembodiment of the present invention generate a strobe composite imagefrom a plurality of frames of a moving image. A first frame is selectedfrom the plurality of frames of the moving image. A plurality of secondframes relating to the first frame are determined. Then, a strobecomposite image is generated by superposing the plurality of secondframes in accordance with a superposing manner.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a schematic block diagram showing the arrangement of an imagecomposition apparatus according to the first embodiment of the presentinvention;

FIG. 2 shows the superposing state of frames;

FIG. 3 shows a strobe composite image generated from a slow movingobject image;

FIG. 4 shows a strobe composite image generated by decimating someframes;

FIG. 5 is a flow chart showing a series of procedures of strobe imagecomposition according to the first embodiment;

FIG. 6 shows an example of an interface which comprises a slider used toselect a point of interest;

FIG. 7 shows an example of a window to be presented to the user todesignate a composition start point;

FIG. 8 is a flow chart showing the procedures for changing a superposingmanner by designating an objective range;

FIG. 9 shows an example of an interface which allows the user to selecttwo frames as start and end points;

FIG. 10 shows a slider operation state on the interface in FIG. 9;

FIG. 11 shows an end point setting state on the interface in FIG. 9;

FIG. 12 shows an interface used to designate a range;

FIG. 13 is a view for explaining range designation by means of theinterface in FIG. 12, i.e., shows a state wherein the point of interestis moved to the start point;

FIG. 14 shows determination of the start point in range designation;

FIG. 15 shows a state wherein the point of interest is moved to the endpoint in range designation;

FIG. 16 shows determination of the end point in range designation;

FIG. 17 shows a state upon setting the second start point in rangedesignation;

FIG. 18 shows a state after the second start point is set in rangedesignation;

FIG. 19 shows selection of the second end point in range designation;

FIG. 20 shows an example of storage data corresponding to the setupcontents of the superposing manner;

FIGS. 21A to 21C show modification examples of the superposing manner;

FIG. 22 is a view for explaining a preview presentation interfaceaccording to the third embodiment of the present invention, i.e., showsa series of input video frames;

FIG. 23 shows one frame to be manually extracted;

FIG. 24 shows the manual extraction result of the frame shown in FIG.23;

FIG. 25 shows a generation example of a strobe composite image includingthe frame shown in FIG. 24;

FIG. 26 shows an example of a preview presentation interface;

FIG. 27 shows a state after a modification process on the previewpresentation interface;

FIG. 28 is a flow chart of a strobe image composition method accordingto the fourth embodiment of the present invention;

FIG. 29 is a flow chart of a strobe image composition method accordingto the fifth embodiment of the present invention;

FIG. 30 shows an example of a user interface used to make the userselect a desired strobe composite image;

FIG. 31 is a flow chart of a strobe image composition method accordingto the sixth embodiment of the present invention;

FIG. 32 shows an example of a process for automatically extracting aregion from an image for answer data;

FIG. 33 is a schematic block diagram showing the arrangement of a strobeimage composition apparatus according to the seventh embodiment of thepresent invention;

FIG. 34 is a flow chart of a strobe image composition method accordingto the eighth embodiment of the present invention;

FIG. 35 is a flow chart of a strobe image composition method accordingto the ninth embodiment of the present invention;

FIG. 36 is a flow chart of a strobe image composition method accordingto the 10th embodiment of the present invention;

FIG. 37 is a flow chart of a strobe image composition method accordingto the 11th embodiment of the present invention;

FIGS. 38A and 38B show examples of feature point tracing results;

FIGS. 39A and 39B show the execution state of a motion patterndetermination step; and

FIG. 40 is a flow chart showing the flow of processes of a strobe imagecomposition method according to the 12th embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will be described hereinafter withreference to the accompanying drawings.

(First Embodiment)

FIG. 1 is a schematic block diagram showing the arrangement of an imagecomposition apparatus according to the first embodiment of the presentinvention. As shown in FIG. 1, an image composition apparatus of thisembodiment comprises a display unit 1 which displays an input video,various windows used to generate and edit and a strobe composite imagebased on the input video, and the like, a central processing unit (CPU)2, an input unit 3 which comprises a keyboard, pointing device (mouse),and the like, and a main storage unit 4. A program 5 used to generateand edit a strobe image, input video data 6, and generated strobe imagedata 7 are recorded in an external storage device such as a hard diskdevice, magnetooptical disk device, or the like.

The image composition apparatus of this embodiment can be implemented assoftware which runs on a versatile computer. In this case, the computerprogram 5 that implements the process for generating and editing astrobe image is read out onto the main storage unit 4, and is executedby the CPU 2. Note that the arrangement of this embodiment incorporatesan operating system (OS) that controls hardware of the computer andprovides a file system and graphical user interface (GUI). In sucharrangement, the computer program which implements the process forgenerating and editing a strobe image is installed as applicationsoftware which runs on the operating system.

Prior to a detailed description of the processing operation of the imagecomposition apparatus according to this embodiment, a schematic sequenceof strobe composition will be explained. In strobe composition, an inputimage is a moving image, and an output image is a strobe composite image(moving or still image). In order to generate an output strobe compositeimage from the input moving image, an object extraction step ofextracting only object portions which are to undergo strobe compositionfrom the input image, and a step of designating a strobe compositionobject that represents frames and a manner of superposing the frames soas to implement strobe composition are required. These two steps areindependent from each other, and either of these steps can be executedfirst.

Let X be an input moving image of an object. Then, strobe compositeimage Y is generated as follows. That is, the first frame of movingimage Y is determined as an image of that of moving image X, and thesecond frame of moving image Y is generated by superposing the secondframe of moving image X on the first frame of moving image Y. The thirdframe of moving image Y is generated by superposing the third frame ofmoving image X on the second frame of moving image Y. Furthermore, the(k+1)-th (k is a natural number) frame of moving image Y is generated bysuperposing the (k+1)-th frame of moving image X on the k-th frame ofmoving image Y. Then, appropriate times are assigned to respectiveframes of moving image Y to generate a moving image. FIG. 2 showsrespective frames of moving image Y.

In this case, frames are overlaid on each other. However, framescorresponding to a user's underlay instruction may be underlaid. Evenwhen only discrete frames are used, as will be described later, framesare overlaid or underlaid in accordance with the time series of inputobject images. The time series of input object images need not be thesame as that upon photographing an object. For example, the reverseorder of the time series, i.e., reverse playback, may be used.

The frames to be superposed need not always be consecutive, but may bediscrete. Especially, the user often wants to select every N-th frames(N is an integer) to be superposed. Hence, a user interface that allowsthe user to select every N-th frames to be superposed from the A-th toB-th frames (A, B, and N are integers) may be provided.

For example, when every other frames are selected as frames to besuperposed like frame Nos. 1, 3, 5, 7, and 9 from a moving imageincluding 10 frames from frame Nos. 0 to 9, a composite image can begenerated from only odd frames of an interlaced-scan moving image.

As another example, when an object moves relatively slowly, if allframes are selected as those to be superposed, a strobe composite imageis complicated (e.g., FIG. 3). In such case, when some frames aredecimated (e.g., every other frames are selected as those to besuperposed), a plain strobe composite image is obtained, as shown inFIG. 4.

An interface which makes the user designate one frame and also thesuperposing manner to composite a strobe image is provided. Using thisinterface, a series of procedures shown in FIG. 5 are executed.Initially, a frame select/input procedure is executed (step S1) to waitfor one frame selected by the user. If the user has selected one frame,a frame shift procedure for calculating and determining an objectiveframe which is commonly called as an IN or OUT point is executed (stepS2). The objective frame obtained by the frame shift procedure undergoesa superposing manner setting procedure (step S3). Finally, a superposingprocedure is executed to obtain a strobe composite image (step S4). Inthe frame select/input procedure (step S1), as shown in, e.g., FIG. 6,an interface that displays a slider 3201 and the currently selectedframe 3202 is presented to the user. On this interface, the user canselect a desired frame by operating the slider 3201 using a mouse orkeyboard.

In the frame select/input procedure, a plurality of frames may bepresented. For example, when the currently selected frame (frame 1402 ofinterest) and frames 1401 and 1403 before and after that frame aredisplayed, as shown in FIG. 7, if the interface is designed to allow theuser to select a frame not only by means of the slider but also byclicking a frame itself (1401 to 1403), the frame selection operationcan become more intuitive and easier to use for the user. Alternatively,frames obtained as a result of the frame shift procedure of thecurrently selected frame are preferably presented. As a result, the usercan immediately recognize frames designated by the currently selectedcontents.

In the superposing manner setting procedure (step S3), for example, theobjective frame is set as one of the start and end points of strobecomposition. When strobe composition is made by superposing Nconsecutive frames (N is a natural number) from a specific start frameto a specific end frame on (or under) the next frames in turn, the startframe is set as the start point of strobe composition, and the end frameis set as the end point. Note that frames need not always beconsecutive, and the start and end points may be similarly determinedfor an arbitrary number of discrete frames. Also, based on user's frameselection, a plurality of frame shift procedures and superposing mannersetting procedures may be simultaneously executed. For example, both thestart and end points of strobe composition may be simultaneouslydetermined based on one designated frame.

A method of designating the start point of strobe composition will bedescribed below with reference to FIG. 7. FIG. 7 shows an example of awindow to be presented to the user to designate the start point. In thisembodiment, assume that M frames from a frame N frames before the frameof interest are to undergo strobe composition, and N and M (integers)are determined in advance. When the user has selected the frame 1402 ofinterest (frame No. 71 in this example), it is determined that overlaystrobe composition is designated to have a frame N frames before theframe of interest (frame No. 65 in this example) as a start frame (startpoint), and a frame M frames after the start frame (frame No. 78 in thisexample) as an end frame (end point). When the user presses a “startpoint determination” button, composition is executed.

Note that execution of strobe composition need not always be started inresponse to depression of the “start point determination” button. Forexample, an interface may be designed to determine that an “overlay”button is pressed while the start to end frames have been selected, andto wait for a composition execution instruction. An image to bepresented may be either an extracted object or an image beforeextraction. In the above example, the end frame is defined. However,only the start point may be selected without determining M.

(Second Embodiment)

An interface which makes the user designate two frames and also thesuperposing manner to implement strobe image composition will beexplained below. Based on this interface, the procedures shown in FIG. 8are executed. In order to quickly obtain a strobe composite image byminimum required user operations, the first embodiment is preferablyused. However, when the user wants to strictly designate the start andend points (frames) of strobe composition, the interface of the secondembodiment is effective. In this embodiment, the superposing manner ofsome frames of the already created strobe composite image can be changedto generate an image again.

The frame select/input procedure is executed to wait for two frames(start and end points) selected by the user in this embodiment (stepS11). If the user has selected the frames, frames between the twoselected frames are determined as those to be processed, and aresuperposed (step S12). In this embodiment, the superposing manner ischanged (step S13), and the superposing procedure is executed again toobtain a strobe composite image (step S14).

An interface which presents a frame select window presents one slider3301, and a frame 3302 to be selected by this slider 3301, as shown in,e.g., FIG. 9, and makes the user select two frames from the slider.Buttons 3303 and 3304 are used to set the current point of interest asthe start and end points, respectively. FIG. 9 illustrates a statewherein the user has pressed the button 3303 to set the point ofinterest (frame No. 45) as the start point. The user moves the slider3301, as shown in FIG. 10, and sets the point of interest (frame No. 71)as the end point using the button 3304, as shown in FIG. 11. As in thefirst embodiment, when the user selects a frame using the slider 3301,not only the currently selected frame but also frames before and afterthat frame may be presented, and the user may select a frame by directlyclicking it in addition to selection using the slider 3301. Thus, theframe selection operation becomes more intuitive and easier to use.

As the superposing manner in step S12, an arbitrary method may be used.For example, frames may be superposed in turn in ascending or descendingorder of time stamp.

A case will be exemplified below wherein a designated range is to besuperposed in association with the series of procedures shown in theflow chart of FIG. 8. FIG. 12 shows an example of a window to bepresented to the user in this case. The interface to be described belowmakes the user select the first frame of objective frames using a button3004 with text, and also select the last frame of the objective framesusing a button 3005 with text. In FIG. 12, reference numerals 3001 and3003 denote fields which display frame images in correspondence with thebuttons 3004 and 3005. With these images, the user can recognize thecontents of the currently selected frames at a glance. A button 3006with text displays the frame number of the current frame of interest,and a field 3002 displays an image corresponding to that frame number.The user can move the point of interest by moving a slider 3013 to theright or left by means of a mouse or keyboard input. Prior to selectionof frames, the user can designate a superposing manner using exclusivepush buttons 3010 and 3011. These buttons are mutually exclusive: whenthe user presses one of these buttons, the other button is restored to anon-pressed state.

FIGS. 13 to 19 show the buttons 3004, 3005, and 3006, and the slider3013 extracted from FIG. 12. A series of operations for selecting thefirst and last frames to be selected will be described below withreference to these drawings. Display objects of the buttons 3004, 3005,and 3006 can display text that indicates button names, and numericalvalues that indicate the frame positions (numbers) (these buttons willbe referred to as “buttons with text” hereinafter). Note that thedisplay contents of a numerical value on each button are updated inaccordance with an operation of the slider 3013 and a settlementinstruction for that button.

When the interface is presented to the user for the first time, neithera start point nor end point are designated at that time. In this state,the end point is grayed out, and its button cannot be pressed, as shownin FIG. 13. The user moves the point of interest to a frame that he orshe wants to set as the start point. The user then presses the button3004 with text of the start point to set the current point of interestas the start point, as shown in FIG. 14.

The user moves the point of interest to a frame that he or she wants toset as the end point, as shown in FIG. 15. At this time, since objectiveframes are not settled, they are indicated in light color. When the userpresses the button 3005 with text of the end point, the end point issettled. The user can make superposing designation of the objectiveframes by pressing one of the exclusive push buttons 3010 and 3011,which are used to designate the superposing manner at that time. Whenthe user presses the button 3005 with text of the end point once againin the state shown in FIG. 16, the settled contents are canceled torevert the state shown in FIG. 15. When the user presses a “compositionexecution” button 3014 in the state shown in FIG. 16, strobe compositionis executed. On the other hand, when the user selects the frame ofinterest again in the state shown in FIG. 16, and presses the startpoint button again, as shown in FIG. 17, he or she can make superposingdesignation of another strobe image for a new period after superposingdesignation made so far, as shown in FIG. 18. The user makes the sameobjective frame selection operations as those described above, i.e.,selects the end point of this new period, as shown in FIG. 19, changesthe depression states of the exclusive push buttons 3010 and 3011 asneeded, and so forth, thus making superposing designation of another setof objective frames.

A button 3007 is used to store the current superposing setup contents inthe storage unit 4. For example, the superposing setup contents indicatesuperposing processes to be applied to frames at relative positions fromthe current point of interest, as shown in FIG. 20. When the userselects the point of interest and presses a button 3012, the savedsuperposing setup contents are loaded and applied. By calling the savedsetup in this way, the edit time can be shortened. Some pre-storeddifferent setups may be prepared as a list, and that list may bepresented in response to a user's request to load another setup. In thisway, a plurality of different setups can be preferably selectively used.

The current superposing setup contents stored in the storage unit 4 maybe permanently recorded using a recording unit such as an HDD or thelike, and may be read out and applied to composition of another strobeimage. That is, the readout superposing setup may be applied to a framegroup, different from the currently processed frame group later, tocomposite another strobe image.

Also, a plurality of superposing setups with different contents arerecorded in the recording unit to simultaneously generate strobecomposite images corresponding to these superposing setups. Furthermore,a plurality of strobe composite images may be generated on the basis ofone arbitrary superposing setup. It is preferable to make onlysuperposing setup processes in advance, and to simultaneously executeimage generation processes later in terms of high work efficiency.

An interface which is used when a strobe composite image is generatedand presented once by overlaying all frames, and is generated againusing objects designated by the user from these frames while changingthe superposing manner to the underlay manner will be described below.Such processes correspond to the procedures in steps S13 and S14 in FIG.8.

As shown in FIGS. 21A to 21C, a composite image is displayed byoverlaying frames (FIG. 21A). In this state, the user selects, e.g.,objects 2101 of three left object images by clicking the left button ofa mouse. The selected objects 2101 are preferably displayed in a colordifferent from other objects (FIG. 21B). When the user clicks the rightbutton of the mouse, the superposing manner of these three selectedobjects 2101 is switched to the underlay manner, and a new compositeimage is displayed, as shown in FIG. 21C. In place of presenting allframes as an overlaid image, the interface shown in FIG. 12 may bepresented in advance. Then, when the user clicks the right button of themouse, overlaid objects may be changed to underlaid objects, and viceversa. In this way, two processes, i.e., designation of a roughsuperposing manner, and adjustment of superposing manners for respectiveframes can be attained within a short period of time.

In this embodiment, the interface which allows the user to selectobjective frames from a strobe composite image, and to additionallydesignate to change their superposing manner has been explained. If aninterface which can also change the superposing order when the userselects objective frames from a strobe composite image is added, thetime required to create and edit a strobe composite image can be furthershortened. Note that changing a strobe composite image at the instanceof clicking requires high processing power of the image compositionapparatus, and also operator's skills to cope with an instantaneouschange in screen. Hence, such process is preferably selectively usedaccording to situations.

In the above description, a frame is selected using the slider.Alternatively, for example, some representative frames may be presentedto prompt the user to select one of these representative frames. Then,only frames near the selected representative frame may be presented toprompt the user to input a superposing instruction. In this way, theaforementioned slider operation can be omitted.

As described above, since the interface which can minimize user'soperations required for strobe composition is provided, the strobecomposition execution sequence can be clarified, and the operation loadon the user can be greatly reduced. Therefore, the time required tocreate a strobe composite image can be shortened, and the usability ofthe image composition system can be improved.

(Third Embodiment)

The third embodiment which comprises an interface that presents apreview during a manual extraction process will be described below withreference to FIGS. 22 to 27. A situation in the middle of a user'smanual extraction process of an object image from input video frames Iato Ic shown in FIG. 22 will be examined. Note that the object image is afish that is swimming from the right to the left on the plane of thedrawing. The user has already extracted object images from the first andsecond frames Ia and Ib. The user is about to extract an object imagefrom the third frame Ic.

As shown in FIG. 23, the user is extracting an object image 1702 so asnot to include an object image 1701. If a portion 1801 of the image isinaccurately extracted, as shown in FIG. 24, its influence appears as anunwanted portion 1901 in a strobe composite image, as shown in FIG. 25,resulting in deterioration of image quality of the strobe compositeimage.

As shown in FIG. 26, in addition to the input image and extracted image,an image 1803 after strobe composition based on a currently extractedimage 1802 is sequentially superposed and is displayed as a preview. Theuser can immediately confirm a portion which is extracted inaccuratelyand adversely influences a strobe composite image, and can easilycorrect such portion to obtain a desired strobe composite image. Asshown in FIG. 27, the user can immediately confirm a corrected strobecomposite image 1804.

Note that “preview” amounts to presenting the processing result based onthe current instruction contents before actual processing, andsequentially updating and presenting a complete image (may be either astill or moving image) in accordance with, e.g., frames to be superposedselected by the user.

Note that not all of the input image, extracted image, and strobecomposite image need always be presented, and only two images, e.g., theinput image and strobe composite image may be presented. Also, a previewmay be presented not only when the user manually extracts an objectimage but also when the user corrects an automatic or semi-automaticextraction result of software.

When an object is extracted from an image region such as a background orthe like, which does not change along with the elapse of time,inaccurate extraction does not influence a strobe composite image, andthe user need not spend much time for such extraction. When a preview ispresented, the user can confirm it. Therefore, the time required togenerate a strobe composite image can be shorted while saving labor.

The user can recognize a final image during an edit process, and neednot check if a strobe composite image is obtained as he or she hadexpected after generation of it. In this way, sequential presentation ofa strobe composite image as a preview greatly contributes to shorteningthe time required to generate the strobe composite image and reductionof the load on the user compared to presentation of an extracted imageas a preview.

(Fourth Embodiment)

FIG. 28 is a flow chart showing a series of processes of strobe imagecomposition according to the fourth embodiment of the present invention.In this embodiment, upon reception of a user's instruction (example:depression of a button, clicking of a mouse button, and the like) duringcapturing a moving image, generation of a strobe composite image startsin response to that instruction as a trigger. An input device is, forexample, a camera which captures a moving image, and is connected to astrobe composition system when it is used. A system that practices thefourth embodiment will be referred to as a 1-instruction type strobecomposition system hereinafter. This system has a queue (first-in,first-out (FIFO) buffer) for saving a moving image. Note that amulti-stage delay circuit may be used in place of the queue.

The 1-instruction type strobe composition system repeats steps S3401 andS3402 shown in FIG. 28 unless an one-click-instruction is detected. Thisinstruction may be done by clicking a mouse or switch by the user, andis referred to as “one-click-instruction”. An image input step (S3401)of capturing the next image frame from the input device, and an imageholding step (S3402) of shifting the queued frames by one frame bydiscarding the first frame in the queue, and adding the captured frameas the last frame are repeated.

Upon detection of an one-click-instruction (S3403: user input step), the1-instruction type strobe composition system sets a timer whichgenerates notification (time out) at least once when a predeterminedperiod of time elapses after the detection time of the instruction(S3403), and proceeds with processes in steps S3401 and S3402. If thetimer generates notification (S3405), the system executes strobecomposition using images saved in the queue (S3406): image compositionstep) and outputs a composite image (S3407: image output step).According to this embodiment, since the user need only input oneinstruction alone, the load on the user can be greatly reduced, and thetime required to generate a strobe composite image can also be greatlyshortened.

(Fifth Embodiment)

FIG. 29 is a flow chart showing a series of processes of strobe imagecomposition according to the fifth embodiment. This embodimentautomatically detects a frame which is important upon executing strobecomposition (to be referred to as a “feature image frame” hereinafter)during capturing of a moving image, and generation of a strobe compositeimage starts in response to that detection as a trigger. Such system ofthe fifth embodiment will be referred to as a “fully automatic strobecomposition system” hereinafter.

As in the fourth embodiment, the fully automatic strobe compositionsystem according to this embodiment has a queue (FIFO buffer) for savinga moving image. This system repeats steps S3501 to S3503 to be describedbelow. That is, the system repeats an image input step (S3501) ofcapturing the next image frame from the input device, an image holdingstep (S3502) of shifting the queued frames by one frame by discardingthe first frame in the queue, and adding the captured frame as the lastframe, and a feature image frame detection step (S3503) of checkingwhether or not a feature image frame is present.

If a feature image frame is present, the fully automatic strobecomposition system sets a timer which generates notification (time out)at least once when a predetermined period of time elapses after thedetection time of the feature frame (S3504), and proceeds with processesin steps S3501 to S3503. If the timer generates notification (S3505),the system executes strobe composition using images saved in the queue(S3506): image composition step) and outputs a composite image (S3507:image output step).

As described above, according to this embodiment, no user's instructionis required, and the time required to generate a strobe composite imagecan be further shortened compared to the fourth embodiment. In thefourth and fifth embodiments that detect a user's instruction or featureimage frame, the timer is used. Alternatively, times may be recorded atgiven time intervals, and an elapse of a predetermined period of timemay be detected. In the fully automatic strobe composition system,various methods of determining a feature image frame and detecting suchframe may be used. Such methods will be explained taking a strobecomposite upon hitting a tee shot at golf as an example. For example,strobe image composition conformity conditions include:

(1) A sound produced upon hitting a tee shot is detected, and an imageframe at the time of detection of the sound is determined as a featureimage frame.

(2) An object shape (template) in a feature image frame is prepared inadvance, an object region is extracted from each image frame, and whenit is determined that the extracted object shape is sufficiently similarto the template, that frame is determined as a feature image frame.

(3) Templates of an object for a feature image frame and several framesbefore and after the feature image frame are prepared in advance, anobject region is extracted from each image frame, and when it isdetermined that the shapes of the object for several successive framesare sufficiently similar to the templates, that frame is determined as afeature image frame.

(4) The area of an object region for each image frame is calculated, thedifference (peak-to-peak value) between the maximum and minimum areas ofseveral successive frames is calculated, and when the peak-to-peak valuehas exceeded a value, which is set separately, a frame that gives themaximum area (or minimum area) is determined as a feature image frame.

Upon extracting an object region, a monochrome background may beprepared in advance, and the chroma-key method that extracts a colorregion different from the background color as an object region may beused. Upon calculating the area of an object region, the object regionmay be extracted using, e.g., the chroma-key method, and the number ofpixels of that region may be calculated as the area. As criteria usedupon determining a similarity with a template image, for example, anarea St of an object region of a template, an area Se of an extractedobject region, and an area Sc of overlapping region between the objectregion of the template and the extracted object region are calculated,and when a smaller one of two values (Sc/St) and (Sc/Se) is equal to orlarger than a threshold value, a high similarity may be determined.

A method suited to an object of target strobe composition is selected inadvance from these methods, and a feature image frame is detected by theselected method, thus improving the detection precision of a featureimage frame. Therefore, the serviceability of the fully automatic strobecomposition system can be improved. As a result, the time required togenerate a strobe composite image can be shortened.

(Present Preset)

The quality of a strobe composite image to be generated by the1-instruction type strobe composition system or fully automatic strobecomposition system depends on the types of input images and the strobecomposition algorithm. For example, when an object region of eachobjective frame is extracted by the chroma-key method, a threshold valuewhich sets a background color range in a color space must be controlled.If the user can set a parameter such as a threshold value in thechroma-key method or the like, high quality can be obtained. However,when the user must set such parameter for each strobe composition, theload on user's operations increases, and the time required for strobecomposition is prolonged. Hence, some different parameters are preparedin advance, and the user can preferably select a parameter to be usedbefore strobe composition. In this manner, the quality of a strobecomposite image can be improved, and high work efficiency can beassured.

(Present Last Frame of Strobe Image or Playback Moving Image)

It is often difficult for some input images to determine a strobecomposition parameter in advance. In this case, parameter candidates arelimited to several types, and strobe composition is made using all thesecandidates. Then, the obtained results are presented to prompt the userto select one of these results. In this way, the user can obtain adesired output image. For example, on a user interface using a mouse anddisplay screen, strobe composition is made using three differentparameters, and the user selects a desired strobe composite image fromlast frames of strobe composition presented on fields 3601, 3602, and3603 by mouse operation, as shown in FIG. 30. In place of the lastframes, strobe composite moving images may be played back. With thismethod, even when sufficiently high quality cannot be achieved by user'sparameter setup operation, the user's operation can be minimized, andthe time required for strobe composition can be consequently shortened.

(Sixth Embodiment)

FIG. 31 is a flow chart of strobe image composition according to thesixth embodiment of the present invention. In this embodiment, automaticextraction parameters used in strobe image composition are determined inadvance. This embodiment will take a sport game as an example, and willexplain a case wherein a strobe image is generated based on capturing animage of a player during that game.

Before the game, a moving image of a player (object) is captured into apersonal computer as answer data (S4101). That moving image is displayedon the display of the personal computer while being frozen frame byframe, and the user accurately inputs a player region of each frameusing a mouse or the like (S4102). Images obtained by extracting theplayer regions are superposed while being overlaid frame by frame, thusgenerating a strobe composite image (S4103). This strobe composite imagehas very high composition quality, and will be referred to as an answerstrobe composite image hereinafter.

A player region is automatically extracted from the answer data image.FIG. 32 shows an example of the extraction method. The differencebetween the luminance values of a frame 4301 to be extracted and anotherreference frame 4302 is calculated. If this inter-frame difference islarger than a threshold value, an object region is determined;otherwise, a background region is determined, thus generating an alphamask 4303. The alpha mask is a bitmap comprising two values, i.e., theobject region and background region.

A plurality of alpha masks 4305 based on differences from otherreference frames 4303 are generated, and their object regions arelocally ANDed to obtain an alpha mask 4306 of an accurate object region.Finally, the obtained alpha mask undergoes profile fitting using amethod described in reference “Profile Fitting based on Self-similarMapping” (Ida, et al., 5th Image Sensing Symposium lecture papers C-15,pp. 115-120, June 1999) or the like, thus obtaining a more accuratealpha mask 4307. Profile fitting is a profile extraction process forobtaining a profile which suffers less errors from a profile whichsuffers many errors.

In case of the aforementioned extraction method according to thisembodiment, extraction parameters include a threshold value used todetermine an object or background region, a frame interval between theobjective frame and reference frame, profile fitting strength, and thelike.

The extraction parameters are temporarily set to certain values, andplayer regions are extracted (S4104). Based on the extraction results ofthe player regions, a temporary strobe composite image is generated(S4105). The temporary strobe composite image normally has poorerquality than the answer strobe composite image. In order to estimate thedegree of quality deterioration, errors of the temporary strobecomposite image from the answer strobe composite image are calculated(S4106). Some combinations of parameters may be determined, and stepsS4104 to S4106 are repeated while switching parameters within the rangeof these combinations. That is, if all predetermined parameters areused, the flow advances to the next step; otherwise, the flow returns tostep S4104 (S4107).

After steps S4104 to S4106 are repeated for all the parameters,parameters which correspond to minimum errors of the temporary strobecomposite image from the answer strobe composite image are formallydetermined as those which are applied to the subsequent process (S4108).The parameters determined in this step will be referred to as executionparameters in this embodiment. Object regions are extracted from imagesother than the answer data images, which are captured during a game(S4109), and a strobe composite image is generated (S4110).

Optimal values of the extraction parameters depend on environmentalchanges such as a player's uniform color, background color, noisefeature produced upon capturing an image by a camera, and the like.However, these environments remain the same during an identical game,and satisfactory composition can be attained without switching parametervalues. According to this embodiment, parameter values which can obtaina satisfactory composite image even during a game can be determined. Inthis manner, automatic composition that does not require any manualoperations can be implemented, and a strobe composite image can begenerated within a short period of time.

Note that a characteristic feature of this embodiment is to evaluateerrors using a composite image in place of an extracted region shape. Asdescribed in the third embodiment, in case of strobe image composition,region shape errors do not influence a composite image. Even when abackground portion is temporarily extracted as a player region, anidentical background image is overwritten on the background portionwhich remains the same, thus obtaining the same composition result asthat obtained when this portion is not detected as a player region. Asthe errors, for example, differences may be calculated betweencorresponding frames, and the sum or square sum of the absolute valuesof the differences may be used.

(Seventh Embodiment)

FIG. 33 is a schematic block diagram showing the arrangement of a strobeimage composition apparatus according to the seventh embodiment of thepresent invention. The seventh embodiment relates to a practicalapparatus which executes a series of processes of strobe imagecomposition that have been explained in the sixth embodiment.

A captured image 4201 is input, and is held in a storage unit 4202 suchas a semiconductor storage element, magnetic recording device, or thelike in case of, e.g., a personal computer. An image 4203 is sent as ananswer data image to a setting unit 4202 which sets an answer objectregion. For example, the image 4203 is displayed on the display of thepersonal computer, and the user accurately inputs an object region usinga mouse or the like. An answer object image 4205 extracted based on theinput region is sent to a composition unit 4206. The composition unit4206 overwrites the object image 4205 in the order of time, and sends ananswer strobe composite image 4207 to a holding unit 4208.

On the other hand, the image 4203 is also sent to an object regiondetection unit 4212. The detection unit 4212 detects an object region onthe basis of extraction parameter values 4211 sent from a parametersetting unit 4210, and an object image 4213 extracted based on thatregion is sent to a composition unit 4214. The composition unit 4212overwrites the object image 4213 in the order of time, and sends astrobe composite image 4215 to an error detection unit 4216. The answerstrobe composite image 4207 is sent from the holding unit 4208 to theerror detection unit 4216. The error detection unit 4216 calculateserrors 4217 between the answer strobe composite image 4207 and strobecomposite image 4215. The calculated errors 4217 are sent to theparameter setting unit 4210.

The parameter setting unit 4210 sequentially switches the parametervalues 4211, checks the result obtained based on the switched parametervalues, and obtains parameter values which can minimize the errors 4217.When another image 4201 is input, the parameter values 4211 are sent tothe detection unit 4212, which detects an object region from an image4203 read out from the storage unit 4202. An object image 4213 undergoesstrobe image composition by the composition unit 4214. A strobecomposite image 4215 obtained as a result of this process is externallyoutput.

With this arrangement, the second and subsequent input images can beautomatically processed, and a strobe composite image can be generatedwithin a short period of time.

(Eighth Embodiment)

The eighth embodiment of the present invention allows the user to selectan image frame used in composition of a strobe image while observing aslow playback video of an object moving image. The eighth embodiment isexecuted from step S3701 in accordance with a series of procedures shownin FIG. 34. Prior to a description of contents executed in respectivesteps, a term “superposing manner switching frame” is used as follows.More specifically, upon generating a strobe composite image bysuperposing a plurality of frames of a moving image, an image frame atwhich a superposing manner (overlay or underlay composition) is switchedwill be referred to as a “superposing manner switching frame”hereinafter. The contents executed in respective steps are as follows.

Step S3701: One image frame is captured and displayed. The flow thenadvances to step S3702.

Step S3702: A user's image frame selection instruction is accepted. Ifno instruction is detected within a predetermined period of time, theflow returns to step S3701. Upon detection of an instruction, the flowadvances to step S3703.

Step S3703: The image frame displayed in step S3701 immediately beforethe user's instruction accepted in step S3702 is determined as asuperposing manner switching frame. A position shifted from the positionof that superposing manner switching frame by a predetermined value inthe time direction is determined as a start point. Also, a positionshifted by another predetermined value is determined as an end point.Then, a predetermined number of image frames at predetermined intervalsof those from the start point to the end point are determined asobjective image frames which are to undergo strobe composition. Afterthat, the flow advances to step S3704.

Step S3704: If image frames to be captured of those which are determinedin step S3703 and are to undergo strobe composition still remain, theseframes are captured. The flow then advances to step S3705.

Step S3705: The image frames which are determined in step S3703 and areto undergo strobe composition are superposed to generate a strobecomposite image. A series of procedures end.

Conventionally, the user cannot often accurately select a desired imageframe while observing a video played back at a frame rate unique to amoving image of an object if he or she makes such attempts. However,according to this embodiment, the user can accurately select a desiredimage frame. As a result, the time required until a strobe compositeimage can be shortened.

Note that this embodiment can be modified as follows.

(1) When the user issues a wrong instruction to be accepted in stepS3702, he or she must quit the series of procedures and redo them tocorrect errors. In order to allow the user to correct such errors, it ispreferable that an image displayed when the user has found errors revertto a previously displayed image, and wrong operations be redoneretrospectively.

(2) In step S3701, an image is displayed. If the user can determine theimage display time, user errors due to an instruction accepted in stepS3702 can be reduced. The display time may be changed during imagedisplay.

(3) One image frame is sequentially captured in step S3701, and arequired number of image frames are also captured in step S3704. Inplace of these processes, image frames may be simultaneously captured,or a plurality of image frames may be captured in step S3701.

(4) The superposing manner switching frame, and the start and end pointsare determined in step S3703 in accordance with the user's instructionaccepted in step S3702. In this case, an image frame corresponding tothe user's instruction is determined as a superposing manner switchingframe. Alternatively, for example, an image frame corresponding to theuser's instruction may be determined as the start or end point.

(5) When the overlay and underlay composition modes need not beswitched, the superposing manner switching frame need not be determined.Hence, in such case, the start and end points may be automaticallydetermined with reference to an image frame corresponding to the user'sinstruction. In this way, the number of times of user's operations canbe reduced, and the time required until a strobe composite image can beshortened.

(6) The superposing manner switching frame, and the start and end pointsare determined in accordance with one user's instruction accepted instep S3702. In order to obtain a strobe composite image that the userwants, one image frame may be determined based on one user's instructionin place of a plurality of image frames. Such modification is effectivewhen a strobe composite image with sufficiently high quality cannot begenerated based on only one user's instruction, and this embodiment canbe applied to more images.

(7) When the start point alone is determined but the end point is notdetermined yet, and when execution of step S3705 starts immediatelyafter the start point is determined, the time required fromdetermination of the end point until completion of strobe compositioncan be shortened.

(Ninth Embodiment)

The ninth embodiment of the present invention allows the user to selectan image frame while observing a playback video of an object strobecomposite moving image. The ninth embodiment is executed according to aseries of procedures shown in FIG. 35. The series of procedures areexecuted from step S3801. The contents executed in respective steps areas follows.

Step S3801: One image frame is captured and displayed. The flow thenadvances to step S3802.

Step S3802. A user's strobe composition start instruction is accepted.If no instruction is detected within a predetermined period of time, theflow returns to step S3801. Upon detection of an instruction, the flowadvances to step S3803.

Step S3803: One frame image is captured. The flow then advances to stepS3804.

Step S3804: A user's instruction which indicates if a superposing manner(overlay or underlay composition) is to be changed upon generating astrobe composite image by superposing image frames is accepted. If achange instruction is detected, the flow advances to step S3805. If noinstruction is detected within a predetermined period of time, the flowjumps to step S3806. Or if an instruction indicating that the method isnot changed is detected, the flow jumps to step S3806.

Step S3805: A superposing manner (overlay or underlay composition) usedupon generating a strobe composite image by superposing image frames ischanged. After that, the flow advances to step S3806.

Step S3806: The image frames are superposed in accordance with thesuperposing manner to generate a strobe composite image. After that, theflow advances to step S3807.

Step S3807: The strobe composite image generated in step S3806 isdisplayed. The flow then advances to step S3808.

Step S3808: A user's instruction indicating if strobe composition is toquit is accepted. If no instruction is detected within a predeterminedperiod of time, the flow returns to step S3803. Or if an instructionindicating that the process does not quit is detected, the flow returnsto step S3803. If a quit instruction is detected, the series ofprocedures quit.

According to this embodiment, since strobe composition, playback of astrobe composite moving image, and selection of image frames can besimultaneously done, the time required until strobe composition that theuser wants is completed can be shortened.

Note that the ninth embodiment can be modified as follows.

(1) When the user issues a wrong instruction to be accepted in stepS3802, S804, or S3808, he or she must quit the series of procedures andredo them to correct errors. In order to allow the user to correct sucherrors, it is preferable that an image displayed when the user has founderrors revert to a previously displayed image, and wrong operations beredone retrospectively.

(2) In steps S3801 and S3807, an image is displayed. If the user candetermine the display time of these images, user errors due toinstructions accepted in steps S3802, S3804, and S3808 can be reduced.The display time may be changed during image display.

(3) One image frame is sequentially captured in each of steps S3801 andS3803. For example, image frames may be simultaneously captured, or aplurality of image frames may be captured in steps S3801 and S3803.

(4) Upon accepting a user's instruction in step S3804, the superposingmanner upon strobe composition is changed. A position shifted from theposition of an image frame corresponding to that user's instruction inthe time direction may be determined as an end point. In this case, whenan image frame corresponding to that end point is superposed to generatea strobe composite image, and that strobe composite image is displayed,the series of procedures end. In this way, one of user's instructionscan be omitted, and the time required until strobe composition iscompleted can be shortened.

(5) The position shifted in the time direction may be determined as astart point in place of the end point. In this way, if image frames atpositions shifted from the position of an image frame corresponding to agiven user's input are determined as those corresponding to the startand end points, and a frame as the boundary upon switching thesuperposing manner, even when the user's instruction accepted in stepS3802 includes wrong contents, operations need not be redone orcorrected, thus shortening the time required for strobe composition.

(6) In the ninth embodiment, a strobe composite image is displayedsimultaneously with strobe composition. The strobe composite image maybe stored in a storage unit, and may be played back later.

(10th Embodiment)

FIG. 36 is a flow chart showing the flow of processes of a strobe imagecomposition method according to the 10th embodiment of the presentinvention. Referring to step S36, S4601 is an image input step ofsequentially inputting time-serial image frames; S4602, an image holdingstep of holding latest N image frames (N is a natural number) input inthe image input step; S4603, a reference image frame detection step ofdetecting a reference image frame serving as a reference for a pluralityof image frames, which are to undergo strobe image composition, on thebasis of strobe image composition conformity conditions used todetermine if a given image frame conforms to strobe image composition;S4604, an image composition step of executing a strobe image compositionprocess for the plurality of image frames stored in the image holdingstep when a predetermined period of time is elapsed after detection ofthe reference image frame; and S4605, an image output step of outputtinga composite image frame that has undergone the strobe image compositionprocess in the image composition step.

The image input step S4601 corresponds to a step of capturing andinputting an object image which is to undergo strobe composition likecamera shooting in a sports live program. In this step, time-serialimage frames are input sequentially. The image holding step S4602 isholding the latest N image frames input in the image input step S4601.For example, frames for latest 2 sec (60 frames at 30 [fps]) of a sportslive video are held while always updating the held contents of a storagemedium with a limited capacity.

The reference image frame detection step S4603 detects a reference imageframe serving as a reference for a plurality of image frames, which areto undergo strobe image composition, on the basis of the strobe imagecomposition conformity conditions used to determine if the currentlyheld image sequence are image frames suited to strobe image compositionwhile the latest N images are held in the image holding step S4602. Forexample, when video frames of a baseball live program are input in theimage input step S4601, the image sequence which is to undergo thestrobe composition process includes a pitching scene of a pitcher, ahitting scene of a batter, and a plurality of image frames of a ball.Note that the reference image frame corresponds to a frame at theinstance when a pitcher releases a ball or a frame at the instance whena batter hits a ball, and a desired strobe composite image can begenerated by superposing frames before and after that reference imageframe. The strobe image composition conformity conditions include, e.g.,a condition using image information, and that using additionalinformation different from image information. The condition using imageinformation uses information only in images. On the other hand, thecondition using additional information different from image informationobtains additional information different from image information inanother input step synchronous with the image input step. In this case,a microphone, infrared sensor, scale, or physical switch is used. Basedon these two conditions, a reference image frame can be detected asfollows.

(Condition Using Image Information)

-   -   A frame from which an image frame similar to an image, which is        registered in advance, is detected is detected as a reference        image frame. For example, the image registered in advance is an        image frame at the instance of impact.    -   A frame from which a change in pixel value of a partial region,        which is set in a frame in advance, is detected is detected as a        reference image frame. For example, when a position where a golf        ball is placed is set as the partial region, a frame in which        white pixels have changed to green pixels is detected as the        reference image frame. Also, a frame when a baseball bat has        passed a given region is detected as the reference image frame.    -   When a mask image similar to a mask image which is set in        advance (a binary image used to separately extract an object        region and background region) is generated, an original image        frame of this mask image is detected as a reference image frame.    -   A frame which has a minimum area of an object region in a mask        image generated from an image frame is detected as a reference        image frame. A hitting scene of baseball or a swing scene of        golf normally has a minimum area of an object region at the        instance of impact.

(Condition Using Additional Information Different from ImageInformation)

-   -   Upon detection of a sound which is set in advance, a frame        corresponding to that detection timing is detected as a        reference image frame. The sound to be detected includes a        hitting sound of baseball, an impact sound of golf, and the        like.    -   When the infrared sensor detects an object to be detected, a        frame corresponding that detection timing is detected as a        reference image frame. For example, an image frame corresponding        to a timing at which the infrared sensor detects that a baseball        bat, pitcher's arm, or golf club has passed a prescribed point,        or a timing at which the sensor detects that a ball placed on a        golf tee or artificial turf mat has disappeared is detected as        the reference image frame.    -   When a change in weight is detected, a frame corresponding to        that detection timing is detected as a reference image frame. A        frame at a timing at which the weight of a golf tee or        artificial turf mat has changed is often suited to a reference        image frame.    -   When the physical switch operates, a frame corresponding to that        operation timing is detected as a reference image frame. For        example, a timing at which a pitcher steps on a pitching plate,        a timing at which a pressure acts on a golf tee or artificial        turf mat, or the like may be detected.

The image composition step S4604 executes strobe image composition ofthe plurality of image frames stored in the image holding step S4602 onthe basis of the reference frame detected in the reference image framedetection step S4603. In this case, the plurality of image frames storedin the image holding step S4602 may undergo strobe compositionimmediately after the reference image frame detection step S4603 detectsa reference image frame. Alternatively, the plurality of image framesstored in the image holding step S4602 may undergo strobe compositionwhen a predetermined period of time elapses after detection of areference image frame. The former process is executed when the referenceimage frame is detected as the last frame of the plurality of frameswhich are to undergo strobe composition. The latter process is executedwhen the reference image frame is detected as a middle one (e.g., theinstance of impact of hitting of baseball or golf swing) of theplurality of frames which are to undergo strobe composition. As thenumber of frames which are to undergo strobe composition, all imageframes held in the image holding step S4602 may be used, or apredetermined number of frames, e.g., (2M+1) frames, i.e., M framesbefore and after the reference image frame, may undergo strobecomposition.

The image output step S4605 outputs a composite image frame that hasundergone the strobe composition process in the image composition stepS4604 as display data or a video file.

Detecting the reference image frame serving as a reference for aplurality of frames, which are to undergo strobe image composition, onthe basis of the strobe image composition conditions used to determinean image frame is suited to storage image composition amounts toautomatic designation of a period of the plurality of image frames. Inthis way, a system which can shorten the time required fromphotographing until a strobe composite video is output, and can output astrobe composite image without the intervention of operator's processescan be realized.

(11th Embodiment)

FIG. 37 is a flow chart of a strobe image composition method accordingto the 11th embodiment of the present invention. Referring to FIG. 37,reference numeral 4701 denotes an input video which is input to generatea strobe composite image, and includes a plurality of image frames thatrepresent an object; 4702, a feature point tracing step of obtaining themoving locus of a feature point of an object; 4703, a motion patterndetermination step of making motion analysis of the object on the basisof the obtained locus pattern; and 4704, a motion pattern determinationresult determined in the motion pattern determination step.

The feature point tracing step 4702 traces a feature point of an objectthat appears in the input video 4701. The user may manually trace thefeature point using a pointing device, or the feature point may beautomatically traced using color information of an image. Upon tracing,for example, a histogram intersection method which generates a colordistribution histogram of a rectangular region around the feature point,and searches for regions having similar color distribution histograms inrespective frames can be used. If an input video has already undergonestrobe composition, a plurality of feature points are superposed on asingle image frame, and the feature point can be traced more easily.FIGS. 38A and 38B show the tracing results of a club head and a golfer'shead top in golf swing as examples of the traced feature points. FIG.38B shows an image around the head of an object shown in FIG. 38A in anenlarged scale, and shows the locus of the golfer's head top.

FIGS. 39A and 39B show the execution state of the motion patterndetermination step. The motion pattern determination step 4703 makesmotion analysis of an object on the basis of the pattern of the featurepoint tracing result as the locus of the feature point obtained in thefeature point tracing step 4702. Note that FIG. 39A shows a case whereinthe motion pattern of the club head is analyzed, and FIG. 39B shows acase wherein the motion pattern of the golfer's head top is analyzed. InFIGS. 39A and 39B, reference numerals 4901 and 4904 denote inputs to themotion pattern determination step 4703, which correspond to the featurepoint tracing results output from the feature point tracing step 4702.Reference numerals 4902 and 4905 denote motion pattern determinationunits each of which determines a motion pattern by comparing the patternof the feature point tracing result with a pattern registered inadvance. Reference numerals 4903 and 4906 denote output examples of themotion analysis results determined by the motion pattern determinationunits 4902 and 4905.

In the locus pattern of the club head of golf, when a player is viewedfrom his or her front side, if the swing locus of the follow-through islarger than that through the downswing, a good, relaxed swing thatexploits the weight of the club head is normally determined. A club headmotion pattern which represents such good swing, that which represents apoor swing, that which represents a swing incident to a beginner, andthe like are registered in advance in the motion pattern determinationunit 4902. One of the registered motion patterns, which is most similarto the input 4901 of the motion pattern, is selected, and the motionanalysis result 4903 is output. In the example of FIG. 39A, the motionanalysis result 4903 indicating “expert” is obtained based on the inputmotion pattern 4901.

As for the locus pattern of the golfer's head top, when a player isviewed from his or her front side, a pattern which suffers lessdeviations in the horizontal and vertical directions is determined as agood swing in which the upper body does not sway and the backbone axisis stable.

In an expert's swing, from a state wherein the weight has shifted to theright foot on the take-back (a state 4907 wherein the locus moves to theleft end in FIG. 39B), the weight shifts to the left foot toward impactat which the golfer hits a ball (the locus moves to the right, asindicated by 4908 in FIG. 39B). Immediately before the impact, a clubhead and arms pull at each other to accelerate the club head (at thistime, the locus of the head top moves to the left in FIG. 39B again),and the impact is reached in a behind-the-ball state 4909. After theimpact, the weight naturally shifts to the left foot (4910). This is themotion pattern of the head top in the expert's swing.

As in the aforementioned club head, a head top motion pattern whichrepresents the expert's swing, that which represents a poor swing, thatwhich represents a swing incident to a beginner, and the like areregistered in advance in the motion pattern determination unit 4902. Oneof the registered motion patterns, which is most similar to the input4901 of the motion pattern, is selected, and the motion analysis result4906 is output. In the example of FIG. 39B, the motion analysis result4906 indicating “expert” is obtained based on the input motion pattern4904.

According to this embodiment described above, in an image compositionmethod that generates a strobe composite image by superposing aplurality of frames of a moving image, the moving locus of a featurepoint of an object is obtained, and the motion analysis of the object ismade on the basis of the obtained locus pattern. Thus, not only a strobecomposite image is visually output, and a motion analysis result thatcan lead to improvement of the athletic ability and the like can beoutput together.

(12th Embodiment)

FIG. 40 is a flow chart of a strobe image composition method accordingto the 12th embodiment of the present invention. Referring to FIG. 40,S5001 is an object region extraction step of extracting an object regionfrom the currently captured image frame in real time; S5002, a referenceframe designation step of designating an image frame which serves as areference upon executing strobe composition; and S5003, an imagecomposition step of making strobe composition using the designatedreference frame and extracted object region.

The object region extraction step S5001 always extracts an object regionin real time irrespective of whether or not an input image is a framewhich is to undergo a strobe composition process. The reference framedesignation step S5002 designates an image frame which serves as areference upon executing strobe composition by, e.g., manual designationof the user or the automatic method described in the 10th embodiment.The image composition step S5003 makes strobe composition using thedesignated reference frame and extracted object region. In the strobeimage composition methods described so far in the above embodiments,after an image frame serving as a reference upon executing strobecomposition is designated, object regions are extracted from a pluralityof frames which are to undergo strobe composition, and strobecomposition is made using these object regions. However, such methodsuffers a problem that the object region extraction processing time isadded to the time required from determination of an output video rangeuntil output of an actual video. However, as described in thisembodiment, when the object region extraction process is always made inreal time, this problem can be solved.

In a strobe image composition method that generates a strobe compositeimage by superposing a plurality of frames of a moving image, thecontrol waits for designation of an image frame serving as a referenceupon executing strobe composition, while an object region is extractedfrom the currently captured image frame in real time. After thereference image frame is designated, strobe composition is made usingthis designated reference image frame, and the extracted object regions.Hence, the time required from photographing until output of a strobecomposite video can be greatly shortened.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. An image composition method for generating a strobe composite imagefrom a plurality of frames of a moving image, the method comprising:selecting a first frame from the plurality of frames of the movingimage; determining a plurality of second frames relating to the firstframe; setting a superposing manner for the strobe composite image; andgenerating the strobe composite image by superposing the plurality ofsecond frames in accordance with the set superposing manner.